Enable sparse file support on macOS - #468
Conversation
|
It's not enabled deliberately due to bugs in macOS found by other projects, see #375 ( |
|
Fair concern; I dug into the references to see what the actual exposure is. The gnulib workarounds boil down to: macOS 12 had broken SEEK_HOLE/SEEK_DATA entirely, and macOS 14 has one specific quirk: SEEK_DATA called from an offset that's already inside data skips that region and returns the next one. Good news: find_sparse_entries_seek only calls SEEK_DATA from hole starts, where that quirk doesn't apply. The one exception is the first call at offset 0. If a file starts with data on macOS 14, that region could silently become a hole. That's the whole risk surface. I probed current macOS (26.4.1, APFS): none of the quirks reproduce, SEEK_DATA(0) returns 0 correctly, and the full suite passes locally and on the CI macOS runner, including the roundtrip checks. I did reproduce your "loses sparseness after close" observation, but that only makes detection fire less often; it can't corrupt anything. Two things would make this safe on older versions too:
Happy to add both to this PR. wdyt? |
2 participants
Sparse file detection in Builder is cfg-gated to Linux, Android, and FreeBSD, so macOS silently falls back to dense copies. The existing implementation already works on macOS unchanged: the non-Linux path probes fpathconf(_PC_MIN_HOLE_SIZE) before using SEEK_HOLE/SEEK_DATA, and APFS supports all three (libc exposes the constants for apple targets).
This PR adds target_os = "macos" to the three cfg gates in src/builder.rs and enables the writing_sparse size assertion on macOS.
I found this while debugging snapshot write amplification in qdrant (qdrant/qdrant#9858). Archiving a 32 MiB hole-backed file with 4 KiB of data via append_path_with_name:
Same code, same file, the gate was the only difference.
Verified on macOS 15 (arm64, APFS): full test suite passes including all 5 sparse tests, and writing_sparse measures 37,888 bytes, the same 4k-block bound as ext4, so the assertion reuses the 37 KiB limit.